1. Field of the Invention
The present invention relates to an image processing apparatus, image sensing apparatus, control method, and recording medium, and particularly to a technique of generating, from raw data after shooting, an image focused on at an arbitrary focal distance.
2. Description of the Related Art
There has recently been proposed a technique of recording the intensity distribution and traveling direction of light as raw data upon shooting, and generating, from the recorded raw data, an image focused on at an arbitrary focal distance in an image sensing apparatus such as a digital camera.
“‘Light Field Photography with a Hand-Held Plenoptic Camera’, Stanford University Computer Science Tech Report CSTR 2005-02 by Ren Ng, et al.” discloses a method (Light Field Photography) of forming images from light beams having passed through different pupil regions of an imaging lens on the respective pixels of an image sensor via a microlens array, thereby splitting and recording light beams entering from various directions. In obtained raw data, adjacent pixels record light beams entering from different directions. By extracting light beams in the same direction from pixels corresponding to respective microlenses, an image shot from this direction can be generated. Further, an arbitrary focal distance is set, and outputs from pixels which record light beams having passed through one point on the focal plane at this focal distance are added. This enables pseudo-generating the pixels of an image focused on at a specific focal distance after shooting.
The method in “Light Field Photography with a Hand-Held Plenoptic Camera” is also applicable to recording of a moving image. More specifically, a moving image focused on at an arbitrary focal distance can be generated from raw moving image data after shooting.
In a moving image, a subject moves during shooting. Hence, in moving image shooting by a general image sensing apparatus, the focal distance of the optical system is changed during shooting in accordance with the movement of a subject to be focused on.
However, when generating a moving image focused on at an arbitrary focal distance from raw moving image data recorded using a method as described in “Light Field Photography with a Hand-Held Plenoptic Camera”, there has not been proposed a method of changing a focal distance in accordance with the movement of a subject to be focused on.